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Executive  Summary 


The  National  Defense  Industrial  Association  (NDIA)  Systems  Engineering  Division  collaborated 
with  the  Institute  of  Electrical  and  Electronic  Engineers  (IEEE)  Aerospace  and  Electronic  Sys¬ 
tems  Society  and  the  Software  Engineering  Institute  (SEI)  of  Carnegie  Mellon®  to  obtain  quanti¬ 
tative  evidence  of  the  benefit  of  systems  engineering  (SE)  best  practices  on  project  performance. 
The  team  developed  and  executed  this  survey  of  system  developers  to  identify  SE  best  practices 
used  on  projects,  collect  performance  data  on  these  projects,  and  search  for  relationships  between 
the  application  of  these  SE  best  practices  and  project  performance. 

The  researchers  surveyed  a  sample  of  system  developers  obtained  through  the  memberships  of  the 
NDIA,  IEEE,  and  International  Council  on  Systems  Engineering  (INCOSE)  using  a  questionnaire 
developed  with  the  SE  expertise  and  the  broadly  diverse  experience  of  research  team  members. 
The  questionnaire  consists  of  three  sections: 

1.  one  to  identify  the  characteristics  of  the  responding  project 

2.  a  second  to  assess  the  project’s  use  of  SE  best  practices 

3.  a  third  to  collect  measures  of  project  performance 

The  survey  was  executed  by  the  SEI  via  the  web.  Policies  ensuring  the  anonymity  of  the  respond¬ 
ents  and  the  confidentiality  of  their  responses  were  enforced  to  protect  the  competition-sensitive 
information  supplied.  Responses  sufficient  for  most  analyses  were  received  from  a  total  of  148 
projects.  Responding  projects  ranged  in  contract  value  from  $100  thousand  to  $20  billion.  U.S. 
defense-related  contracts  accounted  for  69%  of  respondents. 

Responses  were  analyzed  by  the  SEI  to  identify  statistical  relationships  between  the  deployment 
of  SE  best  practices  and  overall  project  and  program  performance.  Summaries  of  the  responses  to 
each  question  in  the  survey  are  presented  in  this  report.  Only  aggregated  responses  are  presented; 
no  information  that  is  traceable  to  any  individual  respondent,  project,  or  organization  is  included. 

The  questiormaire  was  designed  to  assess  a  project’s  use  of  SE  best  practices  by  querying  the  re¬ 
spondent  regarding  the  existence  and  characteristics  of  various  work  products  resulting  from  SE 
processes.  Project  performance  was  assessed  based  on  satisfaction  of  project  cost,  schedule,  and 
technical  goals. 

To  better  understand  the  relationship  between  SE  capability  and  project  performance,  the  ques¬ 
tionnaire’s  assessment  of  SE  capability  addressed  the  project’s  use  of  SE  best  practices  in  1 1 
management  and  technical  process  groups.  As  with  the  relationship  between  total  SE  capability 
and  project  performance,  the  responses  were  analyzed  to  identify  relationships  between  project 
performance  and  the  project’s  use  of  SE  best  practices  in  each  of  the  process  groups.  The  survey 
also  examined  the  relationships  between  project  performance  and  other  factors  such  as  project 
challenge  and  prior  experience. 


Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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The  analysis  of  responses  to  identify  relationships  between  the  project’s  deployment  of  SE  best 
practices  and  the  project’s  performance  is  presented  in  a  companion  report  The  Business  Case  for 
Systems  Engineering  Study:  Results  of  the  SE  Effectiveness  Survey  [Elm  2012b],  which  is  availa¬ 
ble  at  http://www.sei.cmu.edu/library/abstracts/reports/12sr009.cfm. 

To  support  the  companion  results  report,  this  report  provides  details  of  the  survey  responses.  The 
information  contained  in  this  report  constitutes  an  industry  benchmark  of  SE  deployment  that  can 
serve  as  a  point  of  comparison  for  the  processes  and  practices  of  system  developers. 

With  this  knowledge,  system  acquirers  and  system  developers  can  inform  their  judgments  regard¬ 
ing  the  application  of  SE  to  their  projects  and  improve  their  SE  practices  to  further  enhance  pro¬ 
ject  outcomes. 
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Abstract 


This  report  contains  detailed  response  data  from  The  Effectiveness  of  Systems  Engineering:  A 
Survey.  The  survey  had  the  goal  of  quantifying  the  connection  between  the  application  of  systems 
engineering  (SE)  best  practices  to  projects  and  programs  and  the  performance  of  those  projects 
and  programs.  The  survey  population  consisted  of  projects  and  programs  executed  by  system  de¬ 
velopers  reached  through  the  National  Defense  Industrial  Association  (NDIA)  Systems  Engineer¬ 
ing  Division,  the  Institute  of  Electrical  and  Electronics  Engineers  (IEEE)  Aerospace  and  Electron¬ 
ic  Systems  Society,  and  the  International  Council  on  Systems  Engineering  (INCOSE).  Analysis  of 
survey  responses  revealed  strong  statistical  relationships  between  project  performance  and  several 
categories  of  SE  best  practices.  The  survey  results  show  notable  differences  in  the  relationship 
between  SE  best  practices  and  performance  between  more  challenging  and  less  challenging  pro¬ 
jects.  The  statistical  relationship  with  project  performance  is  quite  strong  for  survey  data  of  this 
kind  when  both  SE  capability  and  project  challenge  are  considered  together. 
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1  Introduction 


The  National  Defense  Industrial  Association  (NDIA)  Systems  Engineering  Division,  the  Institute 
of  Electrical  and  Electronics  Engineers  (IEEE)  Aerospace  and  Electronic  Systems  Society,  and 
the  Software  Engineering  Institute  (SEI)  of  Carnegie  Mellon  University  are  collaborating  to  ex¬ 
pand  and  extend  the  2007  NDIA  SE  Effectiveness  Study  [Elm  2008]  to  develop  a  business  case  for 
systems  engineering  (BCSE). 

The  mission  of  this  new  study  is  to  assist  the  SE  community  in  achieving  a  quantifiable  and  per¬ 
sistent  improvement  in  project  performance  through  the  appropriate  application  of  systems  engi¬ 
neering  principles  and  practices.  The  primary  steps  in  the  BCSE  process  are 

1 .  Identify  SE  principles  and  practices  shown  to  provide  benefit  to  project  performance.  This 
activity  is  an  extension  and  a  confirmation  of  the  prior  NDIA  survey. 

2.  Facilitate  the  adoption  of  the  survey  findings  through  the  development  of  tools,  training,  and 
guidance  for  SE  educators,  system  developers,  and  system  acquirers. 

3.  Establish  an  ongoing  means  of  monitoring  and  tracking  the  impact  of  SE  to  enable  continu¬ 
ous  improvement  of  the  SE  framework  and  the  business  case  for  SE,  thereby  driving  contin¬ 
uous  improvement  of  project  results. 

As  part  of  this  new  study,  researchers  developed  a  survey  to  identify  SE  principles  and  practices 
applied  to  development  projects  and  the  performance  achieved  by  those  projects.  The  survey  iden¬ 
tified  the  SE  activities  applied  to  individual  system  development  projects  by  assessing  work  prod¬ 
ucts  resulting  from  specific  SE  activities.  The  survey  assessed  project  performance  in  terms  of 
satisfaction  of  schedule,  budget,  and  technical  requirements.  The  resulting  survey  instmment  is 
included  in  the  appendix. 

The  survey  was  deployed  via  the  internet  to  system  developers  worldwide  through  the  resources 
of  the  NDIA,  IEEE,  and  the  International  Council  on  Systems  Engineering  (INCOSE).  To  en¬ 
courage  trathful  responses,  and  to  protect  confidential  and  proprietary  information,  all  responses 
were  submitted  anonymously  with  no  reference  to  the  respondent,  the  project,  or  the  respondent’s 
organization.  The  research  team  collected  148  complete  responses  between  October  2011  and 
March  2012. 

Responses  were  analyzed  to  assess  the  performance  of  each  project,  the  SE  activities  used  on  the 
project,  the  degree  of  challenge  posed  by  the  project,  and  several  other  factors  that  could  affect 
project  performance.  We  then  analyzed  the  relationships  between  these  various  measures  as  a 
means  of  identifying  the  impact  of  SE  activities.  The  results  of  these  analyses  are  provided  in  the 
companion  report.  The  Business  Case  for  Systems  Engineering:  Results  of  the  SE  Effectiveness 
Survey  [Elm  2012b]. 

That  report  along  with  the  detailed  response  data  available  in  this  report  can  serve  as  an  industry¬ 
wide  benchmark  of  SE  practices  deployed  on  system  development  projects.  This  benchmark  also 
may  be  used  as  a  reference  against  which  the  SE  practices  of  a  company  may  be  gauged. 
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The  questionnaire  contained  the  16  sections  listed  in  Table  3.  The  responses  to  the  questions  in 
each  section  of  the  questionnaire  are  summarized  in  the  subsequent  sections  of  this  report. 


Table  1:  Mapping  of  the  Questionnaire  to  Report  Sections 


Questionnaire 

Section 

Questionnaire  Section  Title 

Report 

Section 

A 

About  this  Project 

2 

B 

About  the  Contract 

3 

C 

About  the  Organization 

4 

D 

Systems  Engineering  -  Project  Planning 

5 

E 

Systems  Engineering  -  Integrated  Product  Teams 

6 

F 

Systems  Engineering  -  Risk  Management 

7 

G 

Systems  Engineering  -  Requirements  Development  and  Management 

8 

H 

Systems  Engineering  -  Trade  Studies 

9 

I 

Systems  Engineering  -  Product  Architecture 

10 

J 

Systems  Engineering  -  Product  Integration 

11 

K 

Systems  Engineering  -  Verification 

12 

L 

Systems  Engineering  -  Validation 

13 

M 

Systems  Engineering  -  Configuration  Management 

14 

N 

Project  Performance  -  Project  Performance 

15 

O 

Project  Performance  -  Other  Performance  Measures 

16 

P 

Conclusion 

17 

CMU/SEI-2012-SR-011  |  2 


2  Questionnaire  Section  A  -  About  this  Project 
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The  project  is  challenging  because  there  is  no 
precedent  for  what  is  being  done. 
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I  CO 

I 

I 

I  lA 

IT 

~ 

Strongly  disagree  Disagree 


Agree 


Strongly  agree 


Figure  1:  Question  A.1  -  Chaiienge  Due  to  Precedent 


This  project  is  challenging  because  significant 
constraints  are  placed  on  the  quality  attributes  of  the 

product. 

80 
60 
40 


0  -I - - - - 

Strongly  disagree  Disagree  Agree  Strongly  agree 

Figure  2:  Question  A. 2  -  Chaiienge  Due  to  Constraints  on  Quaiity  Attributes 
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The  project  is  challenging  because  the  size  of  the 
development  effort  is  large. 


55 


58 


28 


Strongly  disagree  Disagree 


Agree  Strongly  agree 


Figure  3:  Question  A.  3-  Chaiienge  Due  to  Size 


The  project  is  challenging  because  the  technology 
needed  for  this  project  is  not  mature  or  otherwise 
poses  a  high  risk. 


Figure  4:  Question  A. 4  -  Chaiienge  Due  to  Technoiogy 


The  project  is  challenging  because  there  are  extensive 
needs  for  interoperability  with  other  systems. 
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Strongly  agree 

Figure  5:  Question  A. 5-  Chaiienge  Due  to  interoperability 
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The  project  is  challenging  because  there  are 
insufficient  resources  available  to  support  the  project. 


22 


Strongly  disagree  Disagree 


Agree  Strongly  agree 


Figure  6:  Question  A. 6-  Chaiienge  Due  to  insufficient  Resources 


The  project  is  challenging  because  there  are 
insufficient  skills  and  subject  matter  expertise 
available  to  support  the  project. 


Figure  7:  Question  A.7  -  Chaiienge  Due  to  insufficient  Skiiis 
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Table  2:  Question  A.  8-  Other  Sources  of  Challenge 


A.8  The  project  is  challenging  for  other  reasons: 


Technical  challenges  (complexity,  interactions,  harsh  environment,  COTs  utilization,  technology  immaturity,  physi¬ 
cal  constraints,  complex  integration) 

Insufficient  experience  (new  product  area,  new  development  methodology  [agile]) 

Organizational  challenges  (complicated  teaming  arrangements,  geographic  distribution,  multi-national  supply  chain, 
ongoing  reorganizations,  organizational  politics,  subcontractor  management) 

Acquirer  challenges  (lack  of  SE  experience,  poor  requirements  definition,  changing  scope,  insufficient  engagement, 
incompetence,  micro-management,  poor  communications) 

Insufficient  budget  and  schedule 

Engineering  processes  (lack  of  standardized  processes,  lack  of  training) 

Insufficient  SE  (lack  of  PM^  support  for  SE,  lack  of  SE  skills  across  organization) 

Legacy  products  (reverse  engineering,  lack  of  reach-back,  building  on  outdated  products,  building  on  defective 
product  foundation,  defective  GFE^) 

Export  regulations 


In  the  past,  this  project  team  has  successfully 
completed  projects  of  similar  scope. 


84 

33 

24 

- 7 - 

Strongly  disagree 

Disagree 

Agree 

Strongly  agree 

Figure  8:  Question  A.  9-  Project  Team  Past  Success 
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government  furnished  equipment 
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The  requirements  supplied  by  the  customer  for  this 
project  are  well-defined. 


- 72 - 
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Strongly  disagree 

Disagree 
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Strongly  agree 

Figure  9:  Question  A.10-  Weii-Defined  Requirements 


The  requirements  supplied  by  the  customer  for  this 
project  have  not  changed  sufficiently  to  generate  a 
significant  impact  on  the  project. 
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Strongly  agree 

Figure  1 0:  Question  A.11  -  Stabie  Requirements 


What  percentage  of  the  customer  technical 
requirements  were  marked  "TBD"  or  equivalent  at 
time  of  contract  award? 
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Figure  1 1:  Question  A. 12-  Percentage  of  Undefined  Requirements  at  Contract  Award 
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What  percentage  of  the  customer's  technical 
requirements  are  currently  marked  "TBD"  or 
equivalent? 


— 84— 
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Figure  1 2:  Question  A.13-  Percentage  of  Currentiy  Undefined  Requirements 
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Do  you  separately  budget  and  track  Systems 

Engineering  activities? 

124 

20 
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Yes 
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Don't  know 

Figure  13:  Question  A.  14-  Separate  Budget  for  SE  Activities 


Approximately  what  percentage  of  non-recurring 
engineering  (NRE)  does  Systems  Engineering 
represent? 


38 

24 

2 

3^23^21 

0% 

10% 

20% 

30% 

- 1 - 1 - 1 - 1 - 1 - f - 

40%  50%  60%  70%  80%  90%  100% 

Figure  1 4:  Question  A.15-  Percentage  of  Non-Recurring  Engineering 
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How  are  Systems  Engineering  activities  estimated  and 
budgeted? 


They  are  They  are  They  are  Other 

budgeted  as  a  estimated  using  a  estimated  on  a 
percentage  of  the  parametric  cost  task-by-task  basis 
total  model 

development  cost 


Figure  15:  Question  A.16-SE  Estimation  and  Budget  Methods 
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Which  of  the  following  best  describes  the  ultimate 
end-user  of  this  product? 
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Figure  1 6:  Question  A.17-  Type  of  End  User 
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3  Questionnaire  Section  B  -  About  the  Contract 


What  is  the  current  total  contract  value  of  this 
project? 


Mean  4.88E+08 
Std.  Dev.  2.22E+09 
Median  5.05E+07 
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Figure  1 7:  Question  B.1  -  Current  Contract  Vaiue 


What  was  the  initial  contract  value  of  this  project?  Mean  3.95E+08 

Std.  Dev.  1.87E+09 
Median  3.98E+07 
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Figure  18:  Question  B.2-  Initiai  Contract  Vaiue 
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The  change  in  contract  value  is  primarily  due  to 
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43 
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3 

N/A,  No  change 

Change  in  tech, 
scope 

Unplanned 

increases 

- 1 

Other 

Figure  19:  Question  B.3-  Reason  for  Change  in  Contract  Vatue 
Tabie  3:  Question  B.3-  Qther  Sources  of  Contract  Vaiue  Change 


B.3  Other  sources  of  contract  value  chang^ 


There  is  no  development  contract.  There  are  multiple  customers  for  the  end  product  which  is  sold  as  individual/group 
licenses.  The  project  is  not  a  standard  contract  development  effort. 

Customer  funded  the  project  incrementally  due  to  their  budgetary  constraints. 


What  is  the  current  total  planned  duration  of 


Mean 


58.64 


Figure  20:  Question  B.4-  Current  Project  Duration 


Responses  to  this  question  included  more  selections  of  "other.”  Based  on  the  analyst’s  interpretation  of  the 
accompanying  text  explanations,  many  of  these  responses  were  dispositioned  to  the  other  three  categories. 
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What  was  the  initial  total  planned  duration  of 
this  project  or  contract? 


Mean 
Std.  Dev. 
Median 


47.52 

35.36 

38.00 


Figure  21:  Question  B.5-  initiai  Pianned  Project  Duration 


The  change  in  schedule  is  primarily  due  to 


Figure  22:  Question  B.6-  Reason  for  Scheduie  Change'* 


Responses  to  this  question  included  more  selections  of  "other.”  Based  on  the  analyst’s  interpretation  of  the 
accompanying  text  explanations,  many  of  these  responses  were  dispositioned  to  the  other  three  categories. 
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What  was  the  initial  total  budget  for  this  project?  Mean  3.86E+08 

Std.  Dev.  1.85E+09 
Median  3.55E+07 
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Figure  23:  Question  B.7 -  initiai  Totai  Budget 


What  is  the  current  total  budget  for  this  project?  Mean  5.66E+08 

Std.  Dev.  2.35E+09 
Median  5.00E+07 
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Figure  24:  Question  B.8-  Current  Totai  Budget 
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The  change  in  budget  is  primarily  due  to 
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Figure  25:  Question  B.9-  Reason  for  Budget  Change 
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Table  4:  Question  B.9-  Other  Sources  of  Budget  Change 


B.9  Other  sources  of  budget  chang^ 


Fixed  price  contract 


How  many  contract  change  orders  have  been  Mean  123.23 

received?  Std.  Dev.  1023.66 


Figure  26:  Question  B.10-  Number  of  Contract  Change  Orders 


This  contract  includes  provisions  for  additional 
payments  based  on  meeting  or  exceeding  cost, 
schedule,  and/or  performance  targets. 
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Figure  27:  Question  B.11  -  Provisions  for  Additional  Payments 


Responses  to  this  question  included  more  selections  of  "other.”  Based  on  the  analyst’s  interpretation  of  the 
accompanying  text  explanations,  many  of  these  responses  were  dispositioned  to  the  other  three  categories. 
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What  is  the  current  completion  status  Mean  71.01 

of  this  project?  Std.  Dev.  26.53 

-jQ  Median  80.00 
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20%  40%  60%  80%  100% 


58 


37 

16 

8 

Figure  28:  Question  B.12-  Current  Compietion  Status 


What  type  of  contract(s)  was  awarded  for  this  project? 


Figure  29:  Question  B.13-  Type  of  Contract 


Tabte  5:  Question  B.13-  Qther  Contract  Types 


Other  contract  type^ 


Project  is  an  internally  funded  software  product  development.  Contracts  are  for  Individual/group  software  licenses  on 
the  end  product. 


The  contract  has  multiple  CLINs  with  different  contract  types  including  CPAF,  LOE,  and  FFP. 


Three  different  CLINs  include  LOE,  CPFF,  and  FFP. 


Task  Order 


Internal  business  investment  with  some  third  party  R%T  investment 


Internally  governed  project  —  internal  cost  agreement 


Not  contracted 


Responses  to  this  question  included  more  selections  of  "other.”  Based  on  the  analyst’s  interpretation  of  the 
accompanying  text  explanations,  many  of  these  responses  were  dispositioned  to  the  other  two  categories. 
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4  Questionnaire  Section  C  -  About  the  Organization 


This  organization  has  successfully  completed  projects 
similar  in  scope  to  this  one  in  the  past. 


m 

39 

19 

4 

Strongly  disagree  Disagree  Agree  Strongly  agree 


Figure  30:  Question  C.1  -  Previous  Success  with  Simitar  Projects 


Within  this  organization  ... 


Figure  31:  Question  C.2-SE  Qrganization 
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Please  enter  the  country  in  which  most  of  the  design 
and  development  engineering  will  be/was  performed. 

1  /in  _ 1  _ 

120  - 

100  - 

80  - 

60  - 

40  - 

20  - 

®  3  2  2  i  i  i  i 

- 1 - -  - 

U  H 

< 

CO 

Z) 

UK 

South  Africa 

Australia 

Canada 

India 

The 

Netherlands 

Sweden 

Finland 

Figure  33:  Question  C.4-  Location  of  Design  and  Development  Engineering  Work 
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Table  6:  Question  C.5-  Other  Notes  of  Importance  About  the  Project 


Is  anything  else  particularly  important  in  characterizing  your  project,  product,  contract,  or  organi¬ 
zation  within  which  it  resides? 


Security  classification  issues  (lack  of  cleared  staff) 


Organizational  issues  (complex  multi-national  supply  chain,  subcontractors  are  also  competitors,  re-organizations, 
geographic  distribution) 


Acquirer  issues  (multiple  customers,  incomplete  requirements) 


Legacy  products  (defective  GFE) 


Technology  issues  (reliability,  environment,  complexity,  technology  immaturity) 


Skill  and  resource  issues  (inexperienced  developers,  not  enough  SEs,  PM  discourages  SE  use) 


CMU/SEI-2012-SR-011  |  18 


5  Questionnaire  Section  D  -  Project  Pianning 


This  project  utilizes/utilized  a  documented  set  of 
Systems  Engineering  processes  for  the  planning  and 
execution  of  the  project. 


Figure  34:  Question  D.1  -  Use  of  Documented  SE  Processes 


This  project  has/had  an  accurate  and  up-to-date  WBS 
that  included  task  descriptions  and  work  package 
descriptions. 


Figure  35:  Question  D.2-  Use  of  a  WBS^ 


work  breakdown  structure 


CMU/SEI-2012-SR-011  |  19 


This  project  has/had  an  accurate  and  up-to-date  WBS 
that  was  based  upon  the  product  structure. 
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Figure  36:  Question  D.3-  Accuracy  and  Currency  of  the  WBS 
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This  project  has/had  an  accurate  and  up-to-date  WBS 
that  was  developed  with  the  active  participation  of 
those  who  perform  the  systems  engineering  activities. 
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Figure  37:  Question  D.4  -  invoivement  of  Systems  Engineers  in  Maintaining  the  WBS 


This  project  has/had  an  accurate  and  up-to-date  WBS 
that  was  developed  and  maintained  with  the  active 
participation  of  all  relevant  stakeholders. 
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Figure  38:  Question  D.5-  invoivement  of  Stakehoiders  in  Maintaining  the  WBS 
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This  project's  Technical  Approach  is  complete, 
accurate,  and  up-to-date. 


Strongly  disagree  Disagree 


Agree  Strongly  agree 


Figure  39:  Question  D.6-  Quaiity  of  the  Technicai  Approach 


This  project's  Technical  Approach  is  developed  and 
maintained  with  the  active  participation  of  those  who 
perform  the  Systems  Engineering  activities. 
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Figure  40:  Question  D.7  -  Invoivement  of  SE  in  Maintaining  the  Technicai  Approach 


This  project's  Technical  Approach  is  developed  and 
maintained  with  the  active  participation  of  all 
appropriate  functional  stakeholders. 
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Figure  41:  Question  D.8-  Invoivement  of  Stakeholders  in  Maintaining  the  Technical  Approach 
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This  project  has  a  top-level  plan,  such  as  an  IMP,  that 
is  an  event-driven  plan  (i.e.,  each  accomplishment  is 
tied  to  a  key  project  event). 
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Figure  42:  Question  D.9-  Existence  of  a  Top-Levei  Pian 


This  project  has  a  top-level  plan,  such  as  an  IMP,  that 
documents  significant  accomplishments  with  pass/fail 
accomplishment  criteria  for  both  business  and 
technical  elements  of  the  project. 
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Figure  43:  Question  D.10-  Coverage  of  the  Top-Levei  Pian 


This  project  has  a  top-level  plan,  such  as  an  IMP,  that 
is  consistent  with  the  WBS. 
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Figure  44:  Question  D.11  -  Consistency  of  the  Top-Levei  Pian  with  the  WBS 
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This  project  has  an  integrated  event-based  schedule 
that  is  structured  as  a  networked,  multi-layered 
schedule  of  project  tasks  required  to  complete  the 
work  effort. 
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Figure  45:  Question  D.12-  Quaiity  of  the  Scheduie 


This  project  has  an  integrated  event-based  schedule 
that  contains  a  compilation  of  key  technical 
accomplishments  (e.g.,  an  SE  Master  Schedule). 
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Figure  46:  Question  D.13-  Key  Technicai  Accompiishments  in  the  Scheduie 


This  project  has  an  integrated  event-based  schedule 
that  contains  a  compilation  of  key  technical 
accomplishments  (e.g.,  an  SE  Master  Schedule). 
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Figure  47:  Question  D.14-  Measurabie  Basis  for  the  Scheduie 


CMU/SEI-2012-SR-011  |  23 


This  project  has  an  integrated  event-based  schedule 
that  is  consistent  with  the  WBS 


Figure  48:  Question  D.15-  The  Scheduie’s  Consistency  with  the  WBS 


This  project  has  an  integrated  event-based  schedule 
that  identifies  the  critical  path  of  the  program 
schedule. 
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Figure  49:  Question  D.16-  Criticai  Path  in  the  Scheduie 


This  project  has  a  plan  or  plans  for  the  performance  of 
technical  reviews  with  defined  entry  and  exit  criteria 
throughout  the  life  cycle  of  the  project. 
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Figure  50:  Question  D.17-  Pian  for  Periodic  Technicai  Reviews 
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The  SE  function  actively  participates  in  the 
development  and  updates  of  the  project  planning. 
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Figure  51:  Question  D.18-  SE  invoivement  in  Project  Pianning 
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Those  who  perform  SE  activities  actively  participate  in 
tracking/reporting  of  task  progress. 
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Figure  52:  Question  D.19-  SE  Invoivement  in  Tracking  and  Reporting  Progress 


The  acquirer  provided  this  project  with  a  SE  Plan  in  a 
timely  manner. 


Figure  53:  Question  D.20  -  Acquirer-Provided  SE  Plan 
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This  project  has  a  plan  or  plans  that  include  details  of 
the  management  of  the  integrated  technical  effort 
across  the  project  (e.g.,  an  SE  Mgt.  Plan  or  an  SE  Plan). 
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Figure  54:  Question  D.21  -  Pian  for  Integrated  Technical  Effort 


The  SEMP  developed  by  the  project  team  is  aligned 
and  consistent  with  the  SE  Plan  provided  by  the 
acquirer. 
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Figure  55:  Question  D.22  -  SEMP  Consistency  with  the  Acquirer’s  SE  Plan 
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6  Questionnaire  Section  E  -  Integrated  Product  Teams 
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This  project  makes  effective  use  of  IPTs. 
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Figure  56:  Question  E.1  -  Effectiveness  of  iPTs 
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My  acquirer  participates  in  my  IPTs  for  this  project. 
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Figure  57:  Question  E.2  -  Acquirer  Participation  in  IPTs^ 


integrated  product  teams 
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My  suppliers  actively  participate  in  my  IPTs. 


Figure  58:  Question  E.3-  Suppiier  Participation  in  iPTs 


This  project  has  an  IPT  with  assigned  responsibility  for 

SE. 
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Figure  59:  Question  E.4-  iPT  Responsibiiity  for  SE 
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This  project  has  SE  representation  on  each  IPT. 
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Figure  60:  Question  E.5-  SE  Representation  on  iPTs 
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7  Questionnaire  Section  F  -  Risk  Management 


This  project  has  a  Risk  Management  process  that 
creates  and  maintains  an  accurate  and  up-to-date  list 
of  risks  affecting  the  project. 
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Figure  61:  Question  F.1  -  List  of  project  risks 
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This  project  has  a  Risk  Management  process  that 
creates  and  maintains  up-to-date  documentation  of 
risk  mitigation  plans  and  contingency  plans  for 
selected  risks. 
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Figure  62:  Question  F.2-  Risk  Management  and  Contingency  Pians 
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This  project  has  a  Risk  Management  process  that 
monitors  and  reports  the  status  of  risk  mitigation 
activities  and  resources. 
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Figure  63:  Question  F.3-  Risk  Mitigation  Status  and  Resource  Status  Monitoring 


This  project  has  a  Risk  Management  process  that 
assesses  risk  against  achievement  of  an  event-based 
schedule. 


Figure  64:  Question  F.4-  Scheduie  Risk  Assessment 


This  project's  Risk  Management  process  is  integrated 
with  project  decision-making. 
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Figure  65:  Question  F.5-  Risk  Management  in  Decision  Making 
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This  project's  Risk  Management  process  is  integrated 
with  program  cost  and/or  earned  value  management. 


Figure  66:  Question  F.6-  Risk  Management  Integration  with  Cost  Management 


This  project's  Risk  Management  process  is  integrated 
with  program  scheduling. 


Figure  67:  Question  F.7  -  Risk  Management  Integration  with  Program  Scheduling 


This  project's  Risk  Management  process  integrates 
subcontract  or  supplier  risk  management  processes. 


Figure  68:  Question  F.8-  Integration  of  Supplier  Risk  Management  Processes 
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8  Questionnaire  Section  G  -  Requirements  Deveiopment 
and  Management 


This  project  maintains  an  up-to-date  and  accurate 
listing  of  all  requirements  specified  by  the  customer, 
to  include  regulatory,  statutory,  and  certification 
requirements. 
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Figure  69:  Question  G.1  -  Documentation  of  Customer  Requirements 


This  project  maintains  an  up-to-date  and  accurate 
listing  of  all  requirements  derived  from  those 
specified  by  the  customer. 
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Figure  70:  Question  G.2-  Documentation  of  Derived  Requirements 
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This  project  maintains  up-to-date  and  accurate 
documentation  clearly  reflecting  the  hierarchical 
allocation  of  both  customer  and  derived  requirements 
to  each  element  (subsystem,  component,  etc.)  of  the 
system  in  the  configuration  baselines. 


Figure  71:  Question  G.3-  Hierarchicai  Aiiocation  of  Requirements 
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This  project  documents  and  maintains  accurate  and 
up-to-date  descriptions  of  operational  concepts  and 
their  associated  scenarios. 
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Figure  72:  Question  G.4-  Qperationai  Concepts  and  Scenarios 
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This  project  documents  and  maintains  accurate  and 
up-to-date  descriptions  of  use  cases  (or  their 
equivalent). 
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Figure  73:  Question  G.5-  Use  Cases 


This  project  documents  and  maintains  accurate  and 
up-to-date  descriptions  of  product  installation, 
maintenance,  and  support  concepts. 
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Figure  74:  Question  G.6-  instaiiation,  Maintenance,  and  Support  Concepts 


This  project  has  documented  criteria  for  identifying 
authorized  requirements  providers  to  avoid 
requirements  creep  and  volatility. 
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Figure  75:  Question  G.7  -  Criteria  for  Authorizing  Requirements  Providers 
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This  project  has  documented  criteria  (e.g.,  cost 
impact,  schedule  impact,  authorization  of  source, 
contract  scope,  requirement  quality)  for  evaluation 
and  acceptance  of  requirements. 


Figure  76:  Question  G.8-  Criteria  for  Accepting  Requirements 


The  requirements  for  this  project  are  approved  in  a 
formal  and  documented  manner  by  relevant 
stakeholders. 
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Figure  77:  Question  G.9-  Approvai  Process  for  Requirements 


This  project  performs  and  documents  requirements 
impact  assessments  for  proposed  requirements 
changes. 


Figure  78:  Question  G.10-  Requirements  impact  Assessments 
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This  project  develops  and  documents  project 
requirements  based  upon  stakeholder  needs, 
expectations,  and  constraints. 
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Figure  79:  Question  G.11  -  Stakehoider-Based  Requirements 


This  project  has  an  accurate  and  up-to-date 
requirements  management  system. 


Figure  80:  Question  G.12-  Requirements  Management  System 


For  this  project,  the  requirements  documents  are 
managed  under  a  configuration  control  process. 


Figure  81:  Question  G.13-  Configuration  Controi  of  Requirements 
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For  this  project,  the  requirements  documents  are 
accessible  to  all  relevant  project  staff. 


Figure  82:  Question  G.14-  Accessibiiity  of  Requirements 
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9  Questionnaire  Section  H  -  Trade  Studies 


Stakeholders  impacted  by  trade  studies  are  involved  in 
the  development  and  performance  of  those  trade 
studies. 
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Figure  83:  Question  H.1  -  Stakehoiders  Invoivement  in  Trade  Studies 


This  project  performs  and  documents  trade  studies 
between  alternate  solutions  in  a  timely  manner  and  is 
based  upon  definitive  and  documented  selection 
criteria. 


Figure  84:  Question  H.2-  Effectiveness  of  Trade  Studies 
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Documentation  of  trade  studies  is  maintained  in  a 
defined  repository  and  is  accessible  to  all  relevant 
project  staff. 
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Figure  85:  Question  H.3-  Trade  Study  Documentation 
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10  Questionnaire  Section  I  -  Product  Architecture 


This  project  maintains  accurate  and  up-to-date 
descriptions  (e.g.,  interface  control  documents, 
models)  defining  interfaces  in  detail. 
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Figure  86:  Question  1.1  -  Quaiity  of  Interface  Descriptions 


Interface  definition  descriptions  are  maintained  in  a 
designated  location,  under  configuration 
management,  and  accessible  to  all  who  need  them. 
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Figure  87:  Question  1.2-  Management  of  Interface  Descriptions 
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For  this  project,  the  product  high-level  structure  is 
documented,  kept  up  to  date,  and  managed  under 
configuration  control. 
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Figure  88:  Question  i.3-  Maintained  Documentation  of  the  Product  Structure 


For  this  project,  the  product  high-level  structure  is 
documented  using  multiple  views  (e.g.,  functional 
views,  module  views). 


Figure  89:  Question  i.4  -  Muitipie  Views  of  the  Product  Structure 


For  this  project,  the  product  high-level  structure  is 
accessible  to  all  relevant  project  personnel. 
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Figure  90:  Question  i.5-  Accessibiiity  of  Product  Structure  Documentation 
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11  Questionnaire  Section  J  -  Product  Integration 


This  project  has  accurate  and  up-to-date  documents 
defining  its  product  integration  process,  plans,  criteria, 
etc.  throughout  the  life  cycle. 


Figure  91:  Question  J.1  -  Documented  Product  Integration  Process 
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12  Questionnaire  Section  K  -  Verification 


This  project  has  accurate  and  up-to-date  documents 
defining  the  procedures  used  for  the  test  and 
verification  of  systems  and  system  elements. 
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Figure  92:  Question  K.1  -  Documented  Verification  Procedures 


This  project  has  accurate  and  up-to-date  documents 
defining  acceptance  criteria  used  for  the  verification 
of  systems  and  system  elements. 
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Figure  93:  Question  K.2-  Documented  Acceptance  Criteria  for  Verification 
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This  project  has  a  documented  and  practiced  review 
process  for  work  products  that  defines  entry  and  exit 

criteria. 


Figure  94:  Question  K.3-  Documented  and  Practiced  Review  Process 


This  project  has  a  documented  and  practiced  review 
process  that  includes  training  the  reviewers  to 
conduct  reviews. 
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Figure  95:  Question  K.4-  Training  for  the  Review  Process 


This  project  has  a  documented  and  practiced  review 
process  that  defines  criteria  for  the  selection  of  work 
products  for  review. 
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Figure  96:  Question  K.5-  Criteria  for  Seiecting  Work  Products  for  Review 
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This  project  has  a  documented  and  practiced  review 
process  that  tracks  action  items  to  closure. 
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Figure  97:  Question  K.6-  Action  Item  Tracking 


This  project  has  a  documented  and  practiced  review 
process  that  addresses  identified  risks  and  risk 
mitigation  activities  during  reviews. 
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Figure  98:  Question  K.7  -  Inclusion  of  Risk  Management  in  the  Review  Process 


This  project  has  a  documented  and  practiced  review 
process  that  examines  completeness  of  configuration 
baselines. 
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Figure  99:  Question  K.8-  Inclusion  of  Configuration  Baselines  in  the  Review  Process 


CMU/SEI-2012-SR-011  |  45 


This  project  conducts  non-advocate  reviews  and 
documents  results,  issues,  action  items,  risks,  and  risk 
mitigations. 
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Figure  100:  Question  K.9-  Documentation  of  Review  Resuits 
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13  Questionnaire  Section  L  -  Vaiidation 


This  project  has  accurate  and  up-to-date  documents 
defining  the  procedures  used  for  the  validation  of 
systems  and  system  elements. 
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Figure  101:  Question  LI  -  Documented  Vaiidation  Procedures 


This  project  has  accurate  and  up-to-date  documents 
defining  acceptance  criteria  used  for  the  validation  of 
systems  and  system  elements. 
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Figure  102:  L.2  —  Documented  Acceptance  Criteria  for  Vaiidation 
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14  Questionnaire  Section  M  -  Configuration  Management 
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This  project  maintains  a  listing  of  items  managed 
under  configuration  control. 
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Figure  103:  Question  M.1  -  Items  Under  Configuration  Control 


This  project  has  a  configuration  management  system 
that  charters  a  CCB  to  disposition  change  requests. 
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Figure  104:  Question  M.2-  Configuration  Control  Board 


CMU/SEI-2012-SR-011  |  48 


This  project  maintains  records  of  requested  and 
implemented  changes  to  configuration-managed 
items. 
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Figure  105:  Question  M.3-  Change  Records 


This  project  creates  and  manages  configuration 
baselines. 


Figure  106:  Question  M.4-  Configuration  Baseiines 
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15  Questionnaire  Section  N  -  Project  Performance 


This  project  creates  and  manages  cost  and  schedule 
baselines. 


Figure  107:  Question  N.1  -  Cost  and  Scheduie  Basetines 


EVMS  data  are  available  to  decision  makers  in  a  timely 

manner. 


Figure  108:  Question  N.2  -  Avaiiabiiity  ofEVMS^  Data 


Earned  Value  Management  System 


CMU/SEI-2012-SR-011  |  50 


The  requirement  to  track  and  report  EVMS  data  is 
levied  upon  the  project's  suppliers. 
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Figure  109:  Question  N.3-  Requirement  for  Suppiiers  to  Track  and  Report  EVMS  Data 


Variance  thresholds  for  the  CPI  and  SPI  are  defined, 
documented,  and  used  to  determine  when  corrective 
action  is  needed. 


Figure  110:  Question  N.4-  Defined  Variance  Threshoids  for  CPi  and  SPi^° 


EVMS  Cost  Performance  Index  and  Schedule  Performance  Index 
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The  EVMS  is  linked  to  the  technical  effort  through  the 
WBSandthe  IMS. 


Figure  1 1 1:  Question  N.5-  EVMS  Connections  to  the  WBS  and  tMS^^ 


When  is  the  EVMS  baseline  updated? 


Figure  112:  Question  N.6-  Strategy  for  Updating  the  EVMS  Baseiine 


”  work  breakdown  structure  and  integrated  master  schedule 
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What  is  the  projected  cost  variance  at 
completion  for  the  current  contract  baseline? 

45 
40 
35 
30 
25 
20 
15 
10 
5 
0 


Mean  3.69E+06 
Std.  Dev.  1.83E+07 
Median  O.OOE+00 


-TT 


_ 

5 

o 

■uy 

o 

o 

T— 1 

r— 1 

o 

o 

o 

T— 1 

W 

vy 

T— 1 

o 

T— I 

■uy 

1 

w 

1 

w 

1 

w 

+ 

+ 

W 

+ 

T— 1 

■uy 

+ 

Figure  113:  Question  N.7  -  Projected  Cost  Variance 


What  is  the  projected  schedule  variance  at  completion 
for  the  current  contract  baseline? 
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Figure  1 14:  Question  N.8-  Projected  Scheduie  Variance 
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What  is  the  current  cumulative  (or  final)  EVMS  CPI  for 
this  project? 


Figure  1 1 5:  Question  N.9-  Current  EVMS  CPF^ 


What  is  the  current  cumulative  (or  final)  EVMS  SPI  for 
this  project? 
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Figure  1 1 6:  Question  N.10-  Current  EVMS  SPF^ 


EVMS  Cost  Performance  Index 
EVMS  Schedule  Performance  Index 


CMU/SEI-2012-SR-011  |  54 


16  Questionnaire  Section  O  -  Other  Performance  Measures 


What  percentage  of  available  Award  Fees  have  been 
received  by  this  project  in  the  current  period  of 
performance? 
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Figure  117:  Question  0.1  -  Percentage  of  Award  Fees  Coiiected  in  the  Current  Period 


What  percentage  of  available  Award  Fees  have  been 
received  by  this  project  to  date  (i.e.,  in  all  periods)? 


Figure  118:  Question  0.2-  Percentage  of  Award  Fees  Coiiected  During  the  Whoie  Project 
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Requirements  are  being  satisfied  and  remain  on  track 
to  be  satisfied  in  the  product  releases  as  originally 
planned;  they  are  not  being  deleted  or  deferred  to 
later  releases. 
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Figure  119:  Question  0.3-  Satisfaction  of  Requirements 


Overall,  this  project  is  performing  per  the  schedule 
established  in  the  current  IMS  approved  by  the 
acquirer. 
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Figure  120:  Question  0.4-  Compiiance  with  Approved  Schedute 


The  schedule  of  this  project's  critical  path,  when 
compared  to  the  current  IMS  approved  by  the 
acquirer  is  ... 
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Figure  121:  Question  0.5-  Current  Scheduie  Variance 
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This  project  collects  and  tracks  (or  will  collect  and 
track)  reports  of  problems  from  fielded  items. 


Figure  122:  Question  0.6-  Post-fieiding  Probiem  Tracking 


This  project  conducts  (or  will  conduct)  engineering 
assessments  of  all  field  trouble  reports. 


Figure  123:  Question  0.7  -  Engineering  Assessments  of  Probiems 


I  believe  that  my  customer  is  satisfied  with  this 
project's  performance  with  respect  to  the  schedule. 
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Figure  124:  Question  0.8-  Customer  Satisfaction  with  Scheduie 
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I  believe  that  my  customer  is  satisfied  with  this 
project's  performance  with  respect  to  cost. 


Figure  125:  Question  0.9-  Customer  Satisfaction  with  Cost 


I  believe  that  my  customer  is  satisfied  with  this 
project's  performance  with  respect  to  satisfaction  of 
requirements. 
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Figure  126:  Question  0.10-  Customer  Satisfaction  with  Compiiance  to  Requirements 
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Table  7:  Question  0.11  -  Other  Useful  Performance  Indicators 


0.1 1  What  performance  indicators  (beyond  cost  and  schedule)  have  been  particularly  useful  for  manag¬ 

ing  your  project? 

Technical  Performance  Metrics  (TPMs) 

KPP'''  metrics  (%  satisfied,  margins) 

EVMS  metrics  (CPI,  SPI,  ECTC)^^ 

Requirements  metrics  (volatility,  closure,  satisfaction,  trends,  derived  requirements  count) 

Customer  feedback  (comments,  complaints,  report  cards) 

Problem  and  Trouble  Reports  (issue  count,  time  to  resolution) 

Test  metrics  (%  of  tests  passed,  test  defects,  time  to  defect  correction) 

Risk  metrics  (expected  cost  of  risk,  mitigation  status) 

Milestone  metrics  (inchstone  progress,  milestone  accomplishment) 

Award  fee  metrics 

Review  metrics  (peer  review  status,  design  reviews  completed) 

Software  metrics  (progress,  change  rate,  code  growth,  defect  resolution  rate) 

Work  product  metrics  (acceptance  rate,  rejected  CDRLs,  time  to  correct  defect) 

Subcontractor  progress  metrics 

key  performance  parameter 
estimated  cost  to  complete 
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Table  8:  Question  0.12-  Other  Desired  Information  That  Was  Unavailable  to  the  Project 


0.12  What  other  kinds  of  performance-related  information  would  have  been  helpful  for  your  project  or 

program,  but  was  unavailable? 

Software  licensing  delivery  &  tracking 

Formal  debriefs  from  product  user 

Cost  and  effort  estimating  parametrics  earlier  in  the  program 

Realistic  award  fee  scores  based  on  performance  within  our  control 

More  timely  cost  data  —  delays  getting  baseline  established 

Test  progress  vs.  plan 

Customer-provided  metrics  in  cost,  schedule,  risk  management 

Capture  of  performance  statistics  on  unit-level  builds  to  help  establish  statistical  performance  acceptance  criteria 

Participation  in  the  design  and  integration  process,  interaction  with  vendors  and  customers 

Subject  matter  experts  to  monitor  supplier  development 

TPMs^®,  KPPs 

Trend  data  for  getting  work  accomplished 

System  maturity 

User  evaluations 

Initial  stakeholder  feedback  early  in  the  contract 

Specifics  on  CONOPs  and/or  mission  scenarios  from  the  user. 

Personnel  performance 

Better  defined  technical  performance  requirements 

Number  of  active  users,  project  site  usage  statistics 

Automated  drawing  tree  fault  checking 

Greater  insight  into  what  the  customer  hears  from  the  agencies  they  serve 

TPMs  from  associated  subcontractors  were  not  shared  with  us  by  the  customer  but  would  have  been  helpful  in  iden¬ 
tifying  weak  or  risky  areas. 

Visibility  into  the  actual  impact  of  meeting  requirements  on  end  item  system  performance.  Several  tough  require¬ 
ments  were  not  actually  necessary  to  meet  overall  system  performance. 

End  user  system's  performance 

Requirements  and  management  of  business  case  assumptions 

EVMS 

Better  status  and  performance  info  from  subcontractor 

Number  of  issues  with  requirements 

Requirements  S-curves. 

technical  performance  measures 
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Table  9:  Question  0.13-  Other  SE  Effectiveness  Indicators  Currently  Used 


0.13  What  indicators  do  you  use  in  your  project  or  organization  to  determine  Systems  Engineering  ef¬ 

fectiveness? 

Requirements  measures  (count,  volatility,  closure  of  TBDs,  traceability,  waivers,  satisfaction,  burndown,  hours  spent 
per  requirement) 

Risk  measures  (burndown) 

Project  measures  (EVMS,  product  and  CDRL^^  delivery  date  variance,  milestone  completion  variance,  IMS/IMP 
variance,  cost  variance,  schedule  variance,  critical  path  variance,  award  fee  criteria  satisfaction,  headcount) 

Technical  measures  (Technical  Performance  Measures  [TPMs]) 

Test  measures  (verification  plan  variance,  inspection  reports,  defect  detection  profiles,  work  product  inspections, 

QA^°  assessments,  discrepancy  report  data,  rework  measures) 

Customer  measures  (customer  comments,  work  product  acceptance  /  rejection  data,  formal  and  informal  customer 
satisfaction  inquiries,  customer  evaluation  of  SE  efforts) 

Process  measures  (SE  best  practices  reviews,  PM  best  practice  reviews,  process  application  audits,  CMMI  assess¬ 
ments) 

Review  measures  (peer  review  status,  customer  review  status  [e.g.,  SRR,  PDR],  action  item  closures) 

Table  10:  Question  0.14-  Other  SE  Effectiveness  Indicators 

0.14  What  indicators  of  Systems  Engineering  effectiveness  are  regularly  reviewed  across  projects  by  1 

higher  level  management?  1 

Requirements  measures  (count,  volatility,  closure  of  TBDs,  maturity,  traceability,  defects,  satisfaction,  traceability) 

Project  measures  (EVMS,  IMS/IMP  variance,  milestone  completion  variance,  cost  variance,  schedule  variance, 
product  and  CDRL  delivery  variance,  %-complete,  affordability,  headcount) 

Technical  measures  (Technical  performance  measures  [TPMSs],  Key  performance  parameters  [KPPs]) 

Risk  measures  (burndown) 

Customer  satisfaction  measures  (customer  feedback,  customer  comments,  work  product  acceptance  /  rejection 
data) 

Process  measures  (Process  Deviations,  SE  health  assessment,  compliance  with  SE  processes,  process  audits) 

Review  measures  (customer  reviews  [SRR,  PDR,  CDR]'®  completed,  SE  peer  reviews  conducted,  SE  peer  review 
effectiveness,  action  item  closure) 

Test  measures  (inspection  metrics,  work  product  inspection  defects,  verification  status,  rework  measures) 

'''  contract  data  requirements  list 
quality  assurance 

system  requirements  review,  preliminary  design  review,  and  critical  design  review 
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17  Questionnaire  Section  P  -  Conciusion 


Table  1 1:  Question  P.1  -  Other  SE  Comments  About  the  Project  or  the  Survey 


P.1  Is  there  anything  else  that  you  would  like  to  teil  us  about  your  project  or  this  survey? 

The  4  levels  of  choice  are  too  limiting;  there  is  no  ability  to  enter  a  "because"  qualifier  for  a  4-level  choice;  there  is 
no  section  on  "customer-driven  requirements  churn;  my  company  evaluates  requirements  changes  (now  56%)  and 
(lately)  provides  cost/sch. 

Referenced  project  is  huge,  multi-billion,  multi-location,  multi-agency,  multi-nation  contract.  A  Joint  Services  /  Joint 
Country  program  like  this  has  no  precedence.  Schedule  variances  are  due  to  1)  changing  technology,  2)  aggres¬ 
sive  schedule. 

This  project  used  a  concurrent  engineering  approach,  which  proved  to  be  a  differentiator  between  our  success  vs. 
other  programs  that  did  not  fare  as  well. 

Schedule  delay  currently  experienced  is  largely  due  to  slips  in  production  (root  cause  labor  skill  and  capacity  short¬ 
falls)  rather  than  slips  caused  by  design  and  development  difficulties. 

Survey  was  all-inclusive.  This  program  did  especially  well  in  the  survey  because  of  the  customer  interest  and  en¬ 
gagement  in  the  program,  which  began  with  the  initial  RFP.^°  This  was  a  very  experienced  customer  in  systems 
engineering. 

This  project  does  not  fit  well  into  this  survey  since  it  is  not  a  "standard"  contracted  development  effort.  Because  it  is 
an  internally  funded  development,  there  was  no  contract  requiring  submittal  of  planning  documents  so  they  were 
tailored  out  of  the  project. 

Classified,  Best  Effort  Technology  Development  program  with  no  EVMS  since  activities  are  level  of  effort. 

This  project  was  recently  completed,  was  successful,  and  leveraged  into  next  phase. 

Complicated  and  convoluted  contractual  arrangement  make  program  very  hard  to  manage.  Late  or  deficient 

GFI/GFE  and  BFE/BFI  hinders  program  execution  and  adds  cost.  Lack  of  strong  and  sound  systems  engineering  at 
the  prime  contractor  level  impacts  program. 

Contract  is  50%  complete,  but  without  a  definitive  contract  with  the  prime  contractor.  Project  has  received  incre¬ 
mental  (unbudgeted  funding  since  inception).  Proceeding  under  prime  contractor  through  successive  rolling  wave 
baseline  reviews  lasting  ~6  months 

This  contract  has  evolved,  both  in  contract  type  and  role,  over  7  years.  We  were  able  to  re-establish  system  engi¬ 
neering  discipline  over  a  4-prime  contract  Enterprise  that  was  out  of  control  two  years  ago.  Since  then,  the  Enter¬ 
prise  has  become  much  more  stable 

Program  executed  ahead  of  plan  until  environmental  quality  test  failure  occurred  with  Shock  Testing.  Additional 
efforts  associated  with  getting  through  shock  testing  created  cost  and  schedule  over  run. 

We  have  worked  with  iNCOSE  to  try  to  define  useful  Systems  Engineering  Effectiveness  metrics.  None  have  been 
useful  to  date  other  than  requirements  metrics,  which  are  only  a  small  part  of  Systems  Engineering. 

Good  processes  plagued  by  an  unwillingness  to  follow  these,  combined  with  compressed  schedule  and  reduced 
budget  profiles  negotiated  by  Business  Development  negotiations  often  plague  the  program's  effective  systems 
engineering  program  execution. 

Although  cost  and  schedule  increases  were  experienced  on  the  program,  our  original  proposal  cost  was  about 
equal  to  the  final  cost  of  the  project,  and  most  of  the  major  schedule  deliveries  were  met  or  nearly  met. 

This  project  is  driven  by  an  immediate  need  in  the  field  by  the  troops  in  IRAQ  and  is  on  an  extremely  tight  deploy¬ 
ment  schedule  with  the  biggest  success  criteria  based  on  reliability,  endurance,  and  successful  [technical  perfor¬ 
mance]. 

This  particular  contract  was  a  competition  —  two  contractors  executing  in  parallel,  with  a  downselect  to  one.  Project 
did  not  need  to  be  as  difficult  as  it  was.  A  more  amicable,  open  working  relationship  with  the  customer  would  have 
made  things  easier. 

request  for  proposal 
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P.1 


Is  there  anything  else  that  you  would  like  to  teil  us  about  your  project  or  this  survey?  (cont.) 


strong  relationship  allowing  the  customer  to  feel  confident  in  your  ability  to  lead  coupled  with  thorough  understand¬ 
ing  and  execution  of  systems  engineering  process  is  key  to  success. 

Previous  experience  has  indicated  that  the  level  of  systems  engineering  and  performance  tracking  implemented 
should  be  tailored  to  the  size  of  the  project.  In  the  past  our  organization  tried  a  one  size  fits  all  approach  to  critical 
path  management  and 

This  was  an  R&D  project  with  challenging  goals.  Customer  requirements  creep  made  execution  challenging  and 
forced  us,  within  the  budget,  to  spend  efforts  on  lower  priority  issues  at  the  expense  of  higher  priority  ones. 

The  project  was  a  system-of-system  architecture  study,  and  resulted  in  identification  of  a  number  of  preferred  alter¬ 
native  solutions.  Several  sections  of  this  survey  do  not  really  apply  (risk  management,  verification,  validation). 

Project  has  experienced  technical  challenges  in  implementation  and  has  had  to  undergo  a  redesign  to  a  key  com¬ 
ponent,  significantly  impacting  cost  and  schedule 

This  is  a  very  large  program  (>  5M  LLOC)  developed  over  15  years,  and  continues  to  evolve.  Contractor  SE  disci¬ 
pline  and  oversight  waned  several  years  ago  at  customer  direction  (SETA  teams  performed  that  role). 

This  survey  contains  questions  that,  1  believe,  are  not  pertinent  to  the  engineering  effort  and  in  fact,  are  sensitive 
and  company  proprietary. 

1  think  the  survey  should  collect  additional  information  regarding  the  "acquirer"  since  their  interaction,  capabilities 
(or  lack  thereof)  and  involvement  can  have  a  profound  impact  on  the  execution  of  a  program  and  the  effective  utili¬ 
zation  of  resources. 

SE  processes  are  well-defined  and  the  staff  is  trained  on  how  to  apply  them  to  a  program.  Often,  the  program  office 
tries  to  take  short  cuts  as  a  cost-savings  incentive,  by  eliminating  steps  in  the  SE  process,  or  not  funding  SE  tasks 

The  project  is  currently  in  Technology  Development.  My  responses  to  this  survey  encompassed  the  current  [phase] 
and  the  previous  Requirements  Maturation  phase. 

Follow  on  programs  were  needed  and  [executed]  to  get  to  a  useful  level  of  functionality. 

This  is  primarily  a  study  contract. 

This  is  a  joint  service  project.  It  can  be  very  difficult  to  get  multiple  services  to  agree  on  common  requirements.  So 
far,  no  major  issues.  Due  to  project  value  EVMS  system  has  not  been  required. 

The  unplanned  activities  and  cost  increase  encountered  were  attributed  to  our  subcontractor.  Our  subcontractor 
was  also  not  fixed  price.  The  subcontractor  was  involved  in  our  SE  process,  but  their  cost  and  schedule  estimates, 

SE  inputs,  risk  inputs  and  were  inaccurate. 

Some  difficulty  in  differentiating  "agree,”  "strongly  agree,”  "disagree,”  and  "strongly  disagree.”  Would  be  useful  if 
some  examples  were  provided  by  the  survey  as  to  what  the  differentiation  is. 

At  times,  the  answers  to  the  questions  seemed  to  assume  formal  SE  tenets  were  in  place.  Several  times  1  had  to 
estimate  based  on  our  practices. 

This  project  took  marginal  capability  GFE^''  that  was  advertised  as  qualified,  discovered  the  faults,  recovered  to 

CDR  maturity  and  regression  re-qualified.  Additionally  two  major  capabilities  upgrades  were  incorporated  and  re¬ 
gression  tested. 

Development  completed  in  2009.  Program  is  in  production  with  change  orders  issued  against  IDIQ  tables.  Proposal 
for  5  year  contract  extension  anticipated  before  Sept.  2012. 

Award  fee  has  not  been  given  yet. 

Having  a  good  standard  process  doesn't  necessarily  guarantee  success.  There  needs  to  enough  flexibility  in  the 
process  to  work  within  the  customer  boundaries  and  limitations. 

This  project,  though  broad  in  scope,  proves  that  the  design  at  all  levels  can  be  pulled  together  through  strong  sys¬ 
tem  engineering  leadership  at  all  levels.  This  leadership  ensures  that  all  of  the  pieces,  from  all  IPTs,  contribute  to 
meeting  interface  requirements. 

government  furnished  equipment 


CMU/SEI-2012-SR-011  |  63 


P.1  Is  there  anything  else  that  you  would  like  to  tell  us  about  your  project  or  this  survey?  (cont.) 

Project  was  executed  to  an  initial  plan  (based  on  proposal)  which  was  deeply  flawed  as  it  was  generated  by  those 
unfamiliar  with  the  program  challenges.  One  month  in,  everyone  knew  we  had  no  chance  of  meeting  cost  or 
schedule. 

My  project  is  a  mix  of  development  and  with  a  vast  majority  of  the  hours  being  LOE.^^ 

This  program  is  probably  different  compared  to  most  programs.  We  work  from  a  very  simple  SOW^‘‘  and  virtually 
develop  all  system  requirements  on  our  own  and  then  gain  buy-in  from  the  customer  as  opposed  the  customer 
providing  an  SRD^^  upfront  with  crisp  requirements. 

Customer/acquirer  consistently  talks  about  SE  but  never  practices  SE.  Customer  practices  SE  Management  in 
name  and  function  without  understanding  relationship  to  the  execution  of  SE. 

Project  did  not  maintain  an  up-to-date  market  view  regarding  their  deliverables. 

The  scope  was  poorly  understood  at  the  signing  of  the  contract.  There  was  no  understanding  of  what  architecture 
would  be  needed  that  would  be  able  to  meet  the  requirements  at  the  start  of  the  project. 

Well  managed  technically  very  challenging  project 

Project  is  recovering  performance  to  original  baseline  in  a  very  competitive  marketplace.  Competitors  equally 
struggle  to  meet  baseline  performance  requirements. 

Novel  design.  Not  supported  by  prescriptive  requirements.  This  industry  relies  on  prescriptive  requirements. 

Our  schedule  problems  were  mostly  due  to  poor  subcontractor  schedule  performance;  subcontractor  had  an  FFP^® 
subcontract  otherwise  cost  overruns  would  have  been  tremendous. 

This  is  an  internal  product  development  project  where  the  company  (Marketing)  is  the  acquirer.  The  acquirer  is  not 
an  independent  3rd  party  or  government. 

The  program  would  give  one  the  indication  that  performance  is  bad.  Flowever,  breaking  out  the  highly  variable  and 
unpredictable  front-end,  SE  requirements  development  has  been  invaluable.  In  my  opinion,  this  will  result  in  a  pro¬ 
gram  that  will  perform  near  plan. 

Most  core  IPT  members  have  20+  years  of  experience  each  and  the  team  has  good  cohesion,  with  very  little  mem¬ 
ber  turnover  in  the  past  three  years.  The  PM^^  is  extremely  proactive  and  very  results  oriented. 

operations  and  maintenance 

level  of  effort 

statement  of  work 

system  requirements  document 

firm  fixed  price 

program  manager 
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18  Conclusion 


The  results  of  this  study,  as  presented  in  the  report  The  Business  Case  for  Systems  Engineering 
Study:  Results  of  the  SE  Effectiveness  Survey  [Elm  2012b],  clearly  show  that  the  deployment  of 
SE  best  practices  has  a  significant  positive  affect  on  project  performance.  Armed  with  this 
knowledge,  many  organizations  will  want  to  take  steps  to  improve  their  project  performance 
through  deployment  of  improved  SE  practices.  The  first  step  in  such  improvements  is  to  gain  an 
understanding  of  current  SE  practices.  This  understanding  can  be  accomplished  through  using  the 
questionnaire  developed  for  this  study  (see  the  appendix)  to  assess  the  organization’s  projects. 
This  assessment  forms  an  SE  deployment  baseline  for  the  organization. 

The  data  presented  in  this  report  constitute  a  benchmark  for  the  application  of  SE  practices  and 
the  production  of  SE  artifacts  in  the  development  of  systems.  System  developers  may  use  this 
benchmark  as  a  reference  against  which  to  compare  their  SE  deployment  baseline.  Comparison 
between  the  baseline  and  benchmark  data  contained  in  this  report  can  identify  areas  of  strength 
and  weakness.  Process  improvement  efforts  can  then  be  initiated  to  address  identified  weaknesses. 
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Appendix:  Survey  Questionnaire 


SwI^lSl.  Software  Engineering  Institute  |  ( lanio^'iclMel Ion  IEEE 

The  Effectiveness  of  Systems  Engineering: 

A  Survey 

The  National  Defense  Industrial  Association  (NDIA),  the  IEEE  Aerospace  and  Electronic 
Systems  Society  (lEEE-AESS)  and  the  Software  Engineering  Institute  (SEI)  welcome  you 
to  your  personalized  questionnaire  for  our  survey  on  “The  Effectiveness  of  Systems  Engi¬ 
neering.”  Our  hope  is  that  your  participation  will  help  your  project  and  organization  evaluate 
the  effectiveness  of  their  Systems  Engineering  practices  relative  to  the  successes  and  chal¬ 
lenges  reported  by  others  throughout  the  industry. 

Most  of  the  information  necessary  to  complete  the  questionnaire  should  be  easily  accessible 
or  familiar  to  you  or  perhaps  an  informed  designee.  It  should  take  about  30  to  45  minutes  to 
complete  the  questionnaire.  Please  provide  your  best  estimates  if  quantitative  measurements 
are  unavailable. 

Please  complete  the  questionnaire  as  candidly  and  completely  as  you  possibly  can.  The  re¬ 
sults  will  be  useful  to  you,  us  and  others  only  to  the  extent  that  all  survey  participants  do  so. 
There  is  no  need  to  hide  weaknesses  or  embellish  strengths.  Remember  that  your  response 
will  be  anonymous.  Neither  the  SEI  nor  anyone  else  will  know  the  person,  project,  or  organ¬ 
ization  reflected  in  your  response.  The  information,  collected  under  promise  of  non  disclosure 
by  the  SEI,  will  be  held  in  strict  confidence  and  will  not  be  released  in  any  manner.  Survey 
results  will  be  reported  only  in  summary  aggregate  form.  Individual  responses  will  NOT  be 
exposed.  No  attribution  to  people,  projects,  or  organizations  will  be  made. 

A  detailed  summary  report  of  the  survey  results  will  be  prepared  by  the  SEI.  The  report  will 
provide  a  baseline  against  which  you  can  compare  the  performance  of  your  project  and  or¬ 
ganization.  As  a  reward  for  participating  in  this  survey,  the  report  will  be  initially  released  only 
to  those  who  fully  complete  a  survey  questionnaire.  The  report  will  be  not  be  publicly  re¬ 
leased  until  one  year  later. 

Thank  you  once  again  for  your  help  with  this  important  activity.  Please  feel  free  to  contact  us 
at  sei-analvsis@sei.cmu.edu  if  you  have  any  difficulty  with  the  questionnaire. 
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CHARACTERIZATION  OF  THE  PROJECT,  PRODUCT,  CONTRACT,  AND 

ORGANIZATION 


A.  ABOUT  THIS  PROJECT 


The  information  gathered  here  and  in  the  next  few  sections  wiii  be  used  by  the  survey  ana- 
iysts  to  categorize  the  participating  projects  and  organizations  in  order  to  better  understand 
the  responses  to  subsequent  questions  about  systems,  Systems  Enqineerinq  practices,  and 
project  performance. 

The  terms  "Project",  and  "Proqram",  are  used  interchangeabiy  throughout  this  survey.  Both 
refer  to  any  temporary  endeavor,  having  a  defined  beginning  and  end,  undertaken  to  meet 
unique  goais  and  objectives.  Such  endeavors  are  characterized  by  a  defined  set  of  objec¬ 
tives,  a  defined  budget  or  cost  estimate,  and  a  defined  scheduie  or  period  of  performance. 

In  crafting  your  response  to  this  survey,  it  is  important  that  you  keep  in  mind  a  ciear  idea  of 
the  scope  of  the  project  for  which  you  are  responding.  This  wiii  heip  to  ensure  that  your  re¬ 
sponses  regarding  appiied  Systems  Engineering  activities  and  your  responses  regarding  pro¬ 
ject  performance  reiate  to  the  same  body  of  work. 

Foiiowing  are  severai  statements  that  have  been  used  to  characterize  various  deveiopment 
projects.  How  weii  do  the  statements  describe  this  project? 

1 .  The  project  is  chaiienging  because  there  is  no  precedent  for  what  is  being  done.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  is  chaiienging  because  significant  constraints  are  piaced  on  the  quaiity  attrib¬ 
utes  (e.g.  reiiabiiity,  scaiabiiity,  security,  supportabiiity,  etc.)  of  the  product.  (Please  select 
one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  The  project  is  chaiienging  because  the  size  of  the  deveiopment  effort  is  iarge.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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4.  The  project  is  challenging  because  the  technology  needed  for  this  project  is  not  mature  or 
otherwise  poses  a  high  risk.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

5.  The  project  is  challenging  because  there  are  extensive  needs  for  interoperability  with  oth¬ 
er  systems  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

6.  The  project  is  challenging  because  there  are  insufficient  resources  (e.g.  people,  funding) 
available  to  support  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

7.  The  project  is  challenging  because  there  are  insufficient  skills  and  subject  matter  exper¬ 
tise  available  to  support  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

8.  The  project  is  challenging  for  other  reasons  (Please  describe  briefly) 


TI 


9.  In  the  past,  this  project  team  has  successfully  completed  projects  of  similar  scope. 
(Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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10.  The  requirements  supplied  by  the  customer  for  this  project  are  well-defined  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

1 1 .  The  requirements  supplied  by  the  customer  for  this  project  have  not  changed  sufficiently 
to  generate  a  significant  impact  on  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

12.  What  percentage  of  the  customer  technical  requirements  were  marked  “To  Be  Deter¬ 
mined”  or  equivalent  at  time  of  contract  award?  (Please  specify  -  numbers  only,  without 
the  percentage  sign) 

% 

13.  What  percentage  of  the  customer’s  technical  requirements  are  currently  marked  “To  Be 
Determined”  or  equivalent?  (Please  specify  an  approximate  percentage  -  without  the 
percentage  sign) 

% 

14.  Do  you  separately  budget  and  track  Systems  Enqineerinq  activities?  (Please  select  one) 
O  Yes 

O  No 

O  Don't  Know 

15.  Approximately  what  percentage  of  non-recurrinq  enqineerinq  (NRE)  does  Systems  Enqi¬ 
neerinq  represent?  (Please  specify  an  approximate  percentage  -  without  the  percentage 
sign) 

I  % 
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16.  How  are  Systems  Engineering  activities  estimated  and  budgeted?  (Please  check  all  that 
apply) 

□  They  are  budgeted  as  a  percentage  of  the  totai  deveiopment  cost 

□  They  are  estimated  using  a  parametric  cost  modei  (e.g.,  COSYSMO,  SEER,  SLIM, 
TruePianning) 

□  They  are  estimated  on  a  task-by-task  basis 

□  Other  (please  describe) 


TI 


17.  Which  of  the  foiiowing  best  describes  the  uitimate  end-user  of  this  product?  (Please  se¬ 
lect  one) 

O  Government  (USA)  -  defense  reiated 
O  Government  (USA)  -  not  defense  reiated 
O  Government  (non-USA)  -  defense  reiated 
O  Government  (non-USA)  -  not  defense  reiated 
O  Industriai  /  Commerciai 
O  Private  Consumer 
O  Other  (Please  describe) 
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B.  ABOUT  THE  CONTRACT 

1 .  What  is  the  current  total  contract  value  of  this  project?  (Please  specify  in  US  dollars  - 
numbers  only,  without  a  dollar  sign  or  commas) 

I  US  dollars  ($) 

2.  What  was  the  initial  contract  value  of  this  project?  (Please  specify  in  US  dollars  —  num¬ 
bers  only,  without  a  dollar  sign  or  commas) 

I  US  Dollars  ($) 

3.  The  change  in  contract  value  is  primarily  due  to:  (Please  select  one) 

O  Not  applicable;  contract  value  has  not  changed  significantly 

O  Change  in  the  technical  scope  of  the  project 
O  Unplanned  increases  in  the  cost  of  the  project 
O  Other  (please  explain) 

- 3 

4.  What  is  the  current  total  planned  duration  of  this  project  or  contract?  (Please  specify  in 
months  -  numbers  only) 

I  Calendar  months 

5.  What  was  the  initial  total  planned  duration  of  this  project  or  contract?  (Please  specify  in 
months  -  numbers  only) 

I  Calendar  months 

6.  The  change  in  schedule  is  primarily  due  to:  (Please  select  one) 

O  Not  applicable;  schedule  has  not  changed  significantly 

O  Change  in  the  technical  scope  of  the  project 
O  Unplanned  increases  in  the  schedule  for  executing  the  project 
O  Customer  driven  increases  in  the  schedule  for  executing  the  project 
O  Other  (please  explain) 

- 3 

3J  3^ 

7.  What  was  the  initial  total  budget  for  this  project? 

I  US  dollars  ($) 
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8.  What  is  the  current  total  budget  for  this  project? 


I  US  dollars  ($) 

9.  The  change  in  budget  is  primarily  due  to:  (Please  select  one) 

O  Not  applicable;  budget  has  not  changed  significantly 

O  Change  in  the  technical  scope  of  the  project 
O  Unplanned  increases  in  the  cost  of  executing  the  project 
O  Customer  driven  increases  in  the  cost  of  executing  the  project 
O  Other  (please  explain) 

- 3 

3J  3^ 

10.  How  many  contract  change  orders  have  been  received?  (Please  specify  a  number,  ap¬ 
proximate  If  necessary) 

I  Change  orders 

11.  This  contract,  includes  provisions  for  additional  payments  based  on  meeting  or  exceeding 
cost,  schedule,  and/or  performance  targets  (e.g.,  incentive  fees,  award  fees).  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

12.  What  is  the  current  completion  status  of  this  project?  (Please  specify  an  approximate 
percentage  ~  without  the  percentage  sign  ~  e.g.,  60  for  a  project  that  Is  60%  complete) 

I  %  Complete 

13.  What  type  of  contract(s)  was  awarded  for  this  project?  (Please  select  one) 

O  This  is  a  fixed-price  contract  -  the  total  contract  value  is  primarily  determined  by  the  ini¬ 
tial  contract,  (e.g.,  FFP,  FPIF,  FFP-LOE). 

O  This  is  a  cost-reimbursable  contract  -  the  total  contract  value  is  primarily  determined  by 
my  cost  of  executing  the  contract  (e.g.,  CPFF,  CPAF,  CPIF). 

O  This  contract  does  not  fit  the  categories  listed  above.  (Please  describe) 

- 3 

3J  3^ 
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C.  ABOUT  THE  ORGANIZATION 

By  "organization"  we  mean  an  administrative  structure  within  which  (possibiy  many)  projects 
or  simiiar  work  efforts  are  organized  under  common  management  and  poiicies. 

When  thinking  about  your  organization,  piease  answer  for  the  unit  to  which  this  project  re¬ 
ports  administrativeiy,  e.g.,  a  site,  division  or  department,  not  for  a  iarger  enterprise  of  which 
the  organization  to  which  you  report  may  be  a  part. 

Foiiowing  are  severai  statements  that  have  been  used  to  characterize  various  deveiopment 
organizations.  How  weii  do  the  statements  describe  this  project's  parent  organization? 

1.  This  organization  has  successfuiiy  compieted  projects  simiiar  in  scope  to  this  one  in  the 
past.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  Within  this  organization  ...  (Please  select  one) 

O  Systems  Engineering  skiiis  and  responsibiiities  are  contained  in  a  separate  department. 

O  Systems  Engineering  skiiis  and  responsibiiities  are  distributed  throughout  other  de¬ 

partments. 

3.  Which  of  these  best  describes  your  industry  or  service?  (Please  select  one) 

O  Industriai  Manufacturing  and  Services  -  Aerospace  and  Defense 

O  Industriai  Manufacturing  and  Services  -  Eiectronic  and  Eiectricai  Equipment 

O  Industriai  Manufacturing  and  Services  -  Other  (please  specify) 


3 

o 

Transportation 

o 

Energy 

o 

Communications 

o 

Consumer  Goods  and  Services 

o 

Heaith  Care 

o 

Other  (please  specify) 

4.  Piease  enter  the  country  in  which  most  of  the  design  and  deveiopment  engineering  wiii 
be/was  performed. 
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5.  Is  anything  else  particularly  important  in  characterizing  your  project,  product,  contract,  or 
organization  within  which  it  resides.  (Please  describe  here) 


CMU/SEI-2012-SR-011  |  75 


SYSTEMS  ENGINEERING  CAPABILITIES  ASSESSMENT 

This  and  the  next  few  sections  ask  you  about  the  Systems  Engineering  activities  performed 
on  this  project.  Most  of  the  questions  ask  about  the  existence  and  quaiity  of  tangibie  work 
products.  Note  that  the  pertinent  information  often  may  be  distributed  throughout  muitipie 
documents  or  other  work  products;  it  need  not  necessariiy  be  iocated  in  one  particuiar  piace. 

Foiiowing  are  severai  statements  about  work  products  and  activities  that  are  sometimes  used 
for  systems  deveiopment.  Piease  use  the  foiiowing  definitions  to  describe  their  use  on  this 
project: 

Strongly  Disagree:  The  work  product  does  not  exist  or  is  never  used  on  this  project. 
Disagree:  The  work  product  is  of  insufficient  quaiity  or  is  not  used  reguiariy  at  appropriate 
occasions  on  this  project. 

Agree:  The  work  product  or  practice  is  of  good  quaiity  and  it  is  used  reguiariy  on  this  project, 
aithough  not  necessariiy  as  often  as  it  couid  be. 

Strongly  Agree:  The  work  product  or  practice  is  of  exceptionai  quaiity  and  it  is  used  at  neariy 
aii  appropriate  occasions  on  this  project. 


D.  PROJECT  PLANNING 

1 .  This  project  utiiizes/utiiized  a  documented  set  of  Systems  Enqineerinq  processes  for  the 
pianning  and  execution  of  the  project.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has/had  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBSj  that 
inciuded  task  descriptions  and  work  package  descriptions.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  has/had  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBSj  that 
was  based  upon  the  product  structure.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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4.  This  project  has/had  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that 
was  developed  with  the  active  participation  of  those  who  perform  the  systems  engineer¬ 
ing  activities.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

5.  This  project  has/had  an  accurate  and  up-to-date  Work  Breakdown  Structure  (WBS)  that 
was  developed  and  maintained  with  the  active  participation  of  all  relevant  stakeholders 
(e.g.,  developers,  maintainers,  testers,  inspectors,  etc.).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

6.  This  project’s  Technical  Approach  is  complete,  accurate  and  up-to-date.  (Please  select 
one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

7.  This  project’s  Technical  Approach  is  developed  and  maintained  with  the  active  participa¬ 
tion  of  those  who  perform  the  Systems  Engineering  activities.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

8.  This  project’s  Technical  Approach  is  developed  and  maintained  with  the  active  participa¬ 
tion  of  all  appropriate  functional  stakeholders.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

9.  This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  is  an 
event-driven  plan  (i.e.,  each  accomplishment  is  tied  to  a  key  project  event).  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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10.  This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  docu¬ 
ments  significant  accomplishments  with  pass/fail  accomplishment  criteria  for  both  busi¬ 
ness  and  technical  elements  of  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

11.  This  project  has  a  top-level  plan,  such  as  an  Integrated  Master  Plan  (IMP),  that  is  con¬ 
sistent  with  the  Work  Breakdown  Structure  (WBS).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

12.  This  project  has  an  integrated  event-based  schedule  that  is  structured  as  a  networked, 
multi-layered  schedule  of  project  tasks  reguired  to  complete  the  work  effort.  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

13.  This  project  has  an  integrated  event-based  schedule  that  contains  a  compilation  of  key 
technical  accomplishments  (e.g.,  a  Systems  Engineering  Master  Schedule).  (Please  se¬ 
lect  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

14.  This  project  has  an  integrated  event-based  schedule  that  references  measurable  criteria 
(usually  contained  in  the  Integrated  Master  Plan)  reguired  for  successful  completion  of 
key  technical  accomplishments.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

15.  This  project  has  an  integrated  event-based  schedule  that  is  consistent  with  the  Work 
Breakdown  Structure  (WBS).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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16.  This  project  has  an  integrated  event-based  schedule  that  identifies  the  critical  path  of  the 
program  schedule.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

17.  This  project  has  a  plan  or  plans  for  the  performance  of  technical  reviews  with  defined  en¬ 
try  and  exit  criteria  throughout  the  life  cycle  of  the  project.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

1 8.  The  Systems  Engineering  function  actively  participates  in  the  development  and  updates 
of  the  project  planning.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

19.  Those  who  perform  Systems  Engineering  activities  actively  participate  in  track¬ 
ing/reporting  of  task  progress.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

20.  The  acguirer  provided  this  project  with  a  Systems  Engineering  Plan  in  a  timely  manner. 
(Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

21 .  This  project  has  a  plan  or  plans  that  include  details  of  the  management  of  the  integrated 
technical  effort  across  the  project  (e.g.,  a  Systems  Engineering  Management  Plan  or  a 
Systems  Engineering  Plan).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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22.  The  Systems  Engineering  Management  Plan  (or  equivalent)  developed  by  the  project 
team  is  aligned  and  consistent  with  the  Systems  Engineering  Plan),  (or  equivalent)  pro¬ 
vided  by  the  acquirer.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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E.  INTEGRATED  PRODUCT  TEAMS 

1 .  This  project  makes  effective  use  of  integrated  product  teams  (IPTs).  (Please  select  one) 
O  Strongiy  disagree 

O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  My  acquirer  participates  in  my  integrated  product  teams  (IPTs)  for  this  project.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  My  suppiiers  activeiy  participate  in  my  integrated  product  teams  (IPTsj.  (Please  select 
one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  has  an  integrated  product  team  (IPTsj  with  assigned  responsibiiity  for  Sys¬ 
tems  Engineering.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  This  project  has  Systems  Engineering  representation  on  each  integrated  product  teams 
(IPTs).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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F.  RISK  MANAGEMENT 

1 .  This  project  has  a  Risk  Management  process  that  creates  and  maintains  an  accurate  and 
up-to-date  iist  of  risks  affecting  the  project  (e.g.,  risks  to  cost,  risks  to  scheduie,  risks  to 
performance)  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has  a  Risk  Management  process  that  creates  and  maintains  up-to-date  doc¬ 
umentation  of  risk  mitigation  pians  and  contingency  pians  for  seiected  risks  (Please  se¬ 
lect  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  has  a  Risk  Management  process  that  monitors  and  reports  the  status  of  risk 
mitigation  activities  and  resources.  ((Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  has  a  Risk  Management  process  that  assesses  risk  against  achievement  of 
an  event-based  scheduie  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  This  project's  Risk  Management  process  is  integrated  with  project  decision-making. 
(Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

6.  This  project's  Risk  Management  process  is  integrated  with  program  cost  and/or  earned 
vaiue  management.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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7.  This  project's  Risk  Management  process  is  integrated  with  program  scheduiing  (e.g., 
risks  are  incorporated  in  the  program  master  scheduies).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

8.  This  project's  Risk  Management  process  integrates  subcontract  or  suppiier  risk  manage¬ 
ment  processes.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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G.  REQUIREMENTS  DEVELOPMENT  AND  MANAGEMENT 

1.  This  project  maintains  an  up-to-date  and  accurate  iisting  of  aii  requirements  specified  by 
the  customer,  to  inciude  reguiatory,  statutory,  and  certification  requirements.  (Please  se¬ 
lect  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  maintains  an  up-to-date  and  accurate  iisting  of  aii  requirements  derived  from 
those  specified  by  the  customer.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  maintains  up-to-date  and  accurate  documentation  cieariy  reflecting  the  hier- 
archicai  aiiocation  of  both  customer  and  derived  requirements  to  each  eiement  (subsys¬ 
tem,  component,  etc.)  of  the  system  in  the  configuration  baseiines.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  operation- 
ai  concepts  and  their  associated  scenarios.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  use  cases 
(or  their  equivaient).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

6.  This  project  documents  and  maintains  accurate  and  up-to-date  descriptions  of  product 
instaiiation,  maintenance  and  support  concepts.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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7.  This  project  has  documented  criteria  for  identifying  authorized  requirements  providers  to 
avoid  requirements  creep  and  voiatiiity.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

8.  This  project  has  documented  criteria  (e.g.,  cost  impact,  scheduie  impact,  authorization  of 
source,  contract  scope,  requirement  quaiity)  for  evaiuation  and  acceptance  of  require¬ 
ments.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

9.  The  requirements  for  this  project  are  approved  in  a  formai  and  documented  manner  by 
reievant  stakehoiders.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

10.  This  project  performs  and  documents  requirements  impact  assessments  for  proposed 
requirements  changes  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

11.  This  project  deveiops  and  documents  project  requirements  based  upon  stakehoider 
needs,  expectations,  and  constraints.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

12.  This  project  has  an  accurate  and  up-to-date  requirements  manaqement  system.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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13.  For  this  project,  the  requirements  documents  are  managed  under  a  configuration  controi 
process.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

14.  For  this  project,  the  requirements  documents  are  accessibie  to  aii  reievant  project  staff. 
(Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 


CMU/SEI-2012-SR-011  |  86 


H.  TRADE  STUDIES 

I .  Stakeholders  impacted  by  trade  studies  are  involved  in  the  development  and  perfor¬ 
mance  of  those  trade  studies.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

2.  This  project  performs  and  documents  trade  studies  between  alternate  solutions  in  a  time¬ 
ly  manner,  and  based  upon  definitive  and  documented  selection  criteria.  (Please  select 
one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

3.  Documentation  of  trade  studies  is  maintained  in  a  defined  repository  and  is  accessible  to 
all  relevant  project  staff.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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I.  PRODUCT  ARCHITECTURE 

1.  This  project  maintains  accurate  and  up-to-date  descriptions  (e.g.  interface  controi  docu¬ 
ments,  modeis,  etc.)  defining  interfaces  in  detaii.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  Interface  definition  descriptions  are  maintained  in  a  designated  iocation,  under  configura¬ 
tion  management,  and  accessibie  to  aii  who  need  them.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  For  this  project,  the  product  high-ievei  structure  is  documented,  kept  up  to  date,  and 
managed  under  configuration  controi.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  For  this  project,  the  product  high-ievei  structure  is  documented  using  muitipie  views  (e.g. 
functionai  views,  moduie  views,  etc.).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  For  this  project,  the  product  high-ievei  structure  is  accessibie  to  aii  reievant  project  per- 
sonnei.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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J.  PRODUCT  INTEGRATION 

1.  This  project  has  accurate  and  up-to-date  documents  defining  its  product  integration  pro¬ 
cess,  pians,  criteria,  etc.  throughout  the  iife  cycie.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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K.  VERIFICATION 

1 .  This  project  has  accurate  and  up-to-date  documents  defining  the  procedures  used  for  the 
test  and  verification  of  systems  and  system  eiements.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has  accurate  and  up-to-date  documents  defining  acceptance  criteria  used  for 
the  verification  of  systems  and  system  eiements.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  for  work  products  that  defines  entry  and  exit  criteria.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  inciudes  training  the  reviewers  to  conduct  reviews.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

5.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  defines  criteria  for  the  seiection  of  work  products  (e.g.,  requirements 
documents,  test  pians,  system  design  documents,  etc.)  for  review.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

6.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  tracks  action  items  to  ciosure.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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7.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  addresses  identified  risks  and  risk  mitigation  activities  during  reviews. 
(Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

8.  This  project  has  a  documented  and  practiced  review  (e.g.  peer  reviews,  design  reviews, 
etc.)  process  that  examines  compieteness  of  configuration  baseiines.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

9.  This  project  conducts  non-advocate  reviews  (e.g.  reviews  by  quaiified  personnei  with  no 
connection  to  or  stake  in  the  project)  and  documents  resuits,  issues,  action  items,  risks, 
and  risk  mitigations  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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L.  VALIDATION 

1 .  This  project  has  accurate  and  up-to-date  documents  defining  the  procedures  used  for  the 
vaiidation  of  systems  and  system  eiements.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has  accurate  and  up-to-date  documents  defining  acceptance  criteria  used  for 
the  vaiidation  of  systems  and  system  eiements.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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M.  CONFIGURATION  MANAGEMENT 

1.  This  project  maintains  a  iisting  of  items  managed  under  configuration  controi.  (Please 
select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

2.  This  project  has  a  configuration  management  system  that  charters  a  Change  Controi 
Board  to  disposition  change  requests.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

3.  This  project  maintains  records  of  requested  and  impiemented  changes  to  configuration- 
managed  items.  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 

4.  This  project  creates  and  manages  configuration  baseiines  (e.g.,  functionai,  aiiocated, 
product).  (Please  select  one) 

O  Strongiy  disagree 
O  Disagree 
O  Agree 
O  Strongiy  Agree 
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N.  PROJECT  PERFORMANCE:  EARNED  VALUE  MANAGEMENT 

1.  This  project  creates  and  manages  cost  and  schedule  baselines.  (Please  select  one) 

O  Strongly  disagree 

O  Disagree 
O  Agree 
O  Strongly  Agree 

2.  Earned  Value  Management  System  (EVMSj  data  are  available  to  decision  makers  in  a 
timely  manner  (i.e.  current  within  2  weeks).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

3.  The  requirement  to  track  and  report  Earned  Value  Management  System  (EVMS)  data  is 
levied  upon  the  project’s  suppliers.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

4.  Variance  thresholds  for  the  Cost  Performance  Index  (CPI)  and  Schedule  Performance 
Index  (SPI)  are  defined,  documented,  and  used  to  determine  when  corrective  action  is 
needed.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

5.  The  Earned  Value  Management  System  (EVMS)  is  linked  to  the  technical  effort  through 
the  Work  Breakdown  Structure  (WBS),  the  Integrated  Master  Plan  (IMP),  (or  equivalent), 
and  the  Integrated  Master  Schedule  (IMS)  (or  equivalent).  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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6.  When  is  the  Earned  Value  Management  System  (EVMS)  baseline  updated?  (Please  se¬ 
lect  as  many  as  apply) 

□  Only  at  contract  initiation 

□  Whenever  a  contract  change  order  or  renewal  is  received 

□  Incrementally  in  rolling  wave  planning 

□  Whenever  the  project  is  reprogrammed  due  to  a  pre-determined  cost  or  schedule  vari¬ 
ance 

□  At  periodic  intervals 

□  Other  (Please  describe  briefly) 


:3 


7.  What  is  the  projected  cost  variance  at  completion  for  the  current  contract  baseline? 
(Please  specify  an  amount  In  US  Dollars  ($),  using  +  signs  for  any  overruns  and  -  signs 
for  any  underruns) 


US  Dollars  ($) 


8.  What  is  the  projected  schedule  variance  at  completion  for  the  current  contract  baseline? 
(Please  specify  In  months,  using  +  signs  for  any  late  delivery  and  -  signs  for  early  deliv¬ 
ery) 


Duration  in  months 


9.  What  is  the  current  cumulative  (or  final)  EVMS  Cost  Performance  Index  (CPI)  for  this  pro¬ 
ject?  (Please  specify  a  number) 


10.  What  is  the  current  cumulative  (or  final)  EVMS  Schedule  Performance  Index  (SPI)  for  this 
project?  (Please  specify  a  number) 
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O.  OTHER  PERFORMANCE  INDICATORS 

1 .  What  percentage  of  available  Award  Fees  have  been  received  by  this  project  in  the  cur¬ 
rent  period  of  performance?  (Please  specify  an  approximate  percentage  -  without  the 
percentage  sign.  Enter  "n/a"  if  this  contract  does  not  include  Award  Fees.) 


2.  What  percentage  of  available  Award  Fees  have  been  received  by  this  project  to  date  (i.e., 
in  all  periods)?  (Please  specify  an  approximate  percentage  -  without  the  percentage 
sign.  Enter  "n/a"  if  this  contract  does  not  include  Award  Fees.) 


3.  Requirements  are  being  satisfied  and  remain  on  track  to  be  satisfied  in  the  product  re¬ 
leases  as  originally  planned;  they  are  not  being  deleted  or  deferred  to  later  releases. 
(Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

4.  Overall,  this  project  is  performing  per  the  schedule  established  in  the  current  Integrated 
Master  Schedule  (IMS)  approved  by  the  acquirer.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

5.  The  schedule  of  this  project’s  critical  path,  when  compared  to  the  current  Integrated  Mas¬ 
ter  Schedule  (IMS)  approved  by  the  acquirer  is  ...  (Please  select  one) 

O  Greater  than  6  months  late 
O  3  to  6  months  late 
O  1  to  3  months  late 
O  Within  plus  or  minus  1  month 
O  1  to  3  months  early 
O  3  to  6  months  early 

6.  This  project  collects  and  tracks  (or  will  collect  and  track)  reports  of  problems  from  fielded 
items.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 
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7.  This  project  conducts  (or  will  conduct)  engineering  assessments  of  all  field  trouble  re¬ 
ports.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

8.  I  believe  that  my  customer  is  satisfied  with  this  project's  performance  with  respect  to  the 
schedule.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

9.  I  believe  that  my  customer  is  satisfied  with  this  project's  performance  with  respect  to  cost. 
(Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

10.  I  believe  that  my  customer  is  satisfied  with  this  project's  performance  with  respect  to  sat¬ 
isfaction  of  requirements.  (Please  select  one) 

O  Strongly  disagree 
O  Disagree 
O  Agree 
O  Strongly  Agree 

11.  What  performance  indicators  (beyond  cost  and  schedule)  have  been  particularly  useful 
for  managing  your  project?  (Please  describe  here) 

- 3 

jlU  3^ 

12.  What  other  kinds  of  performance  related  information  would  have  been  helpful  for  your 
project  or  program,  but  was  unavailable?  (Please  describe  here) 


“31 
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13.  What  indicators  do  you  use  in  your  project  or  organization  to  determine  Systems  Engi¬ 
neering  effectiveness?  (Please  describe  here) 


14.  What  indicators  of  Systems  Engineering  effectiveness  are  reguiariy  reviewed  across  pro¬ 
jects  by  higher  ievei  management?  (Please  describe  here) 
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P.  IN  CONCLUSION 

1 .  Is  there  anything  else  that  you  would  like  to  tell  us  about  your  project  or  this  survey? 
(Please  describe  here) 


Thank  you  very  much  for  your  time  and  effort! 


Please  be  sure  to  use  the  Save  button.  That  will  take  you  to  the  final  page  where  you  may 
SUBMIT  your  response. 


Copyright  2012,  Carnegie  Mellon  University. 
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gineers  (IEEE)  Aerospace  and  Electronic  Systems  Society,  and  the  International  Council  on  Systems  Engineering  (INCOSE).  Analysis  of 
survey  responses  revealed  strong  statistical  relationships  between  project  performance  and  several  categories  of  SE  best  practices.  The 
survey  results  show  notable  differences  in  the  relationship  between  SE  best  practices  and  performance  between  more  challenging  and 
less  challenging  projects.  The  statistical  relationship  with  project  performance  is  quite  strong  for  survey  data  of  this  kind  when  both  SE 
capability  and  project  challenge  are  considered  together. 
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